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Introduction 


Impetus 

The  38th  Chief  of  Staff  (CSA)  GEN  Odierno  directed  the  Center  for  Army  Leadership  (CAL)  to  provide  a 
one-time  demonstration  of  a  developmental  assessment  center  for  strategic  non-commissioned  officer 
(NCO)  positions,  specifically  the  nominative  command  sergeant  major  (CSM)  role.  This  tasking  was 
prompted  by  findings  from  the  2014  National  Defense  Authorization  Act  and  CSA  Way  Point  #2  (Adaptive 
Army  Leaders  for  a  Complex  World,  CSA  Strategic  Priorities,  JAN  14),  which  recommended  expanding 
developmental  assessment  events  to  increase  leader  awareness  of  current  leadership  skills  and  abilities 
as  measured  against  future  leader  position  requirements.  The  leader  assessment  center  concept  was 
then  adopted  as  a  part  of  the  Army  Leader  Development  Program  (ALDP)  as  initiative  1-13-003,  Army 
Priority  List  (APL)  #6R. 

Assessment  Center  Methodology 

The  assessment  center  methodology  has  been  widely  used  in  the  private  sector  to  develop 
individuals  by  using  a  structured  and  objective  series  of  assessments  and  exercises  that  measure  leader 
skills  relative  to  higher  level  roles.  An  assessment  center  is  defined  as  a  standardized  evaluation  of 
behavior  based  on  multiple  inputs.  Several  trained  observers,  referred  to  as  assessors,  and  techniques 
are  used.  These  trained  assessors  then  make  judgments  about  behavior  are  made  from  assessments 
and/or  simulations  specifically  developed  to  fit  the  demands  of  the  target  role  or  level.  These  judgments 
are  pooled  in  a  meeting  among  the  assessors  or  by  a  statistical  integration  process,  which  are  then  used 
for  administrative  purposes  (e.g.  selection,  promotion,  etc.)  or  developmental  feedback  to  the  participant 
(International  Task  Force  on  Assessment  Center  Guidelines,  2009). 

Assessment  centers  were  first  used  by  the  German  Army  in  the  early  twentieth  century.  The  Office  of 
Strategic  Services  in  the  U.S.  first  used  assessment  centers  in  during  World  War  II  to  select  espionage 
agents  (Smith,  1 972).  The  assessment  center  methodology  has  been  used  by  the  U.S.  Army  for  selection 
to  Special  Forces  and  to  ranger  units  for  decades.  The  commercial  sector  uses  the  assessment  center 
method  to  develop  individuals  by  using  a  structured  and  objective  series  of  assessments  and  exercises 
that  measure  leader  skills  relative  to  higher-level  roles.  The  intent  is  to  train  and  select  mid-level 
managers  and  senior  executives.  An  assessment  center  is  defined  as  a  standardized  evaluation  of 
behavior  based  on  multiple  inputs.  Several  trained  observers  and  techniques  are  used.  Judgments  about 
behavior  are  made  from  specifically  developed  assessments.  These  judgments  are  pooled  in  a  meeting 
among  the  assessors  or  by  following  a  process  that  integrates  scores  (International  Task  Force  on 
Assessment  Center  Guidelines,  2009). 

There  is  substantial  evidence  from  commercial  entities  and  the  Army  that  increased  performance  and 
readiness  follow  from  assessment  center  events  (Gaugler,  et.  al,  1 987;  Engelbrecht  &  Fischer,  1 995; 
Schmidt  &  Hunter,  1998.  A  leading  commercial  firm  that  specializes  in  development  using  an  assessment 
center  methodology  found  that  senior  managers  who  participated  were  twice  as  likely  to  be  promoted 
compared  to  a  matched  sample  of  employees  who  did  not  participate  (Byham,  2005).  Participants  were 
also  more  likely  to  be  actively  working  their  individual  development  plan  a  year  after  being  assessed  and 
that  developmental  activities  improved  (Byham,  2005).  Byham  (2005)  found  that  using  the  assessment 
center  method  for  selection  improved  participant  mindsets  from  an  entitlement  mentality  to  a  greater 
understanding  of  the  importance  of  merit-based  promotion  decisions,  and  led  to  a  fundamental  change  in 
how  the  business  recruited,  hired,  developed,  and  managed  its  people.  Meta-analytic  studies  of 
assessment  centers  have  also  produced  the  criterion-related  validity  evidence  of  both  overall  ratings  as 
well  as  separate  dimension  ratings  (Arthur,  Day,  McNelly,  &  Edens,  2003). 

Overview  and  Purpose 

The  CSM  Leader  Development  Assessment  Center  (LDAC)  was  a  demonstration  of  the  assessment 
center  methodology.  The  purpose  of  this  assessment  center  was  to  provide  feedback  to  CSMs  on  their 
leadership  capabilities  for  strategic  NCO  positions,  specifically  the  nominative  CSM  role.  The  LDAC 
provided  a  rigorous  and  purposeful  process  to  assess  the  skills  of  senior  NCO  leaders  and  accelerate 
their  development  for  success  at  the  strategic  leader  level.  A  key  component  of  the  assessment  center 
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methodology  is  that  it  is  based  on  input  from  multiple  measures  (i.e.,  simulations,  tests,  and  interviews)  in 
a  standardized  process.  The  integrated  results  of  these  different  assessments  provide  an  indication  of 
what  leaders  are  capable  of  doing  and  their  potential  to  be  successful  in  senior  strategic  roles.  This  LDAC 
demonstration  was  not  for  administrative  purposes,  like  promotion  decisions  or  selection  for  course 
attendance;  it  was  only  used  to  facilitate  self-awareness  and  development  for  participating  leaders. 

CSMs  and  sergeants  major  (SGMs)  attending  the  Command  Sergeants  Major  Developmental  Program 
(CSMDP)  at  the  School  for  Command  Preparation  (SCP)  were  selected  to  participate  in  the  assessment 
center  demonstration.  The  LDAC  activities  occurred  from  17-23  October  2014  and  were  designed  not  to 
interfere  with  the  standard  CSMDP  curriculum. 

LDAC  Development 

The  development  and  execution  of  the  LDAC  was  performed  over  six  months.  The  key  developers  of  this 
LDAC  demonstration  were  CAL  research  psychologists  holding  doctoral  degrees  in  human  factors, 
industrial/organizational,  and  cognitive/social  psychology.  All  were  certified  in  the  protection  of  human 
subjects  in  research  and  experts  in  the  design  and  development  of  psychological  assessments  of  work 
behaviors  and  individual  differences.  The  LDAC  development  and  demonstration  process  had  five  key 
stages,  which  are  covered  in  this  report  (see  Figure  1  for  development  timeline): 

■  Competency  Model  Development  and  Validation 

■  Scenario  Development  and  Test  Selection 

■  Training  and  Calibration 

■  LDAC  Administration  &  Execution 

■  After  Action  Review 


Figure  1 :  LDAC  Timeline 
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Nominative  CSM  Competency  Model  Development 

The  development  of  the  competency  model  used  for  the  LDAC  was  led  by  a  team  of  personnel 
psychologists  from  the  Leadership  Research,  Assessment,  and  Doctrine  Division  at  CAL.  Personnel  or 
industrial/organizational  (I/O)  psychologists  have  specialized  education  and  knowledge  in  human 
behavior  in  work  settings  and  in  applying  that  knowledge  to  training  and  development,  testing, 
performance  measurement,  job  design,  assessment,  selection,  and  placement.  The  development  process 
for  the  assessment  center  demonstration  went  through  two  key  stages:  initial  research  and  review  from 
subject  matter  experts  (SMEs).  At  the  core  of  the  development  process  was  the  Army’s  concept  of 
leadership  as  defined  in  ADRP  6-22  (Department  of  the  Army,  201 2).  Army  doctrine  defines  leadership 
activities  to  contain  three  basic  goals:  to  lead  others,  to  develop  the  organization  and  its  individual 
members,  and  to  accomplish  the  mission  (i.e.  Leads,  Develops,  and  Achieves).  These  goals  are 
extensions  of  the  Army’s  strategic  goal  of  remaining  relevant  and  ready  through  effective  leadership.  The 
leadership  requirements  model  outlines  the  attributes  and  competencies  Army  leaders  must  develop  to 
meet  these  goals.  The  competencies  and  attributes  outlined  in  ADRP  6-22  are  the  foundation  on  which 
the  specific  requirements  of  the  nominative  CSM  position  have  been  nested. 

Initial  Research  and  SME  Review 

The  focal  role  of  the  LDAC  was  on  the  nominative  CSM  position,  which  is  defined  as  “any  authorized 
CSM  or  SGM  billet  where  the  rated  CSM/SGM  is  rated  by  a  General  Officer  (GO)  or  Senior  Executive 
Agent  (SES).  The  position  must  be  validated  on  an  MTOE  or  TDA  or  a  provisional  organization  approved 
by  HQDA.”  (Sergeant  Major  of  the  Army,  201 1).  The  foundational  research  on  the  nominative  CSM  role 
came  from  a  study  conducted  in  2013-14  by  the  Sergeants  Major  Management  Office  and  the  Institute  for 
Noncommissioned  Officer  Professional  Development  (INCOPD).  That  research  was  conducted  as  a 
needs  analysis  to  assist  in  curriculum  development  for  a  new  nominative  CSM  course.  The  research 
consisted  of  interviews  and  surveys  of  current  nominative  CSMs  and  their  respective  general  officers  on 
the  perceived  strengths  and  potential  skill  gaps  in  the  nominative  role.  The  data  obtained  from  these  two 
sources  were  used  to  create  the  first  draft  of  the  competency  model  and  behaviors  to  be  assessed  in  the 
LDAC.  They  identified  the  key  demands  of  the  strategic,  nominative  CSM  role.  This  initial  model  was 
comprised  of  seven  competencies  organized  around  four  factors. 

This  initial  competency  model  for  LDAC  was  then  reviewed  by  experts  from  the  following  organizations: 
INCOPD,  the  Center  for  Strategic  Leadership  and  Development  at  the  Army  War  College,  the  School  for 
Command  Preparation,  and  the  Army  Research  Institute  (ARI).  Concurrent  with  this  review,  the  model 
was  validated  by  SMEs  who  were  surveyed  on  the  criticality  of  each  competency  as  well  as  necessary 
additions  or  revisions  to  the  specific  behaviors.  This  competency  model  survey  was  sent  to  17  currently 
serving  nominative  CSMs,  ranging  from  the  1  -  to  3-star  level,  identified  by  the  Mission  Command  Center 
of  Excellence  (MCCoE)  headquarters.  Responses  were  received  from  seven  SMEs  which  provided  both 
validation  and  revision  guidance  for  the  model.  Subsequent  revisions  were  made  to  the  nominative  CSM 
requirements  model,  which  was  broken  down  into  two  components,  Intellect  and  Advising,  and  five  key 
competencies  (see  Table  1).  See  Appendix  A  for  a  more  detailed  definition  of  each  competency. 
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Table  1 :  Final  Nominative  CSM  Competency  Model _ 

INTELLECT 

Judgment/Decisionmaking 

The  capacity  to  assess  situations  shrewdly  and  draw  sound  conclusions  and  opinions,  make  sensible 
decisions  and  reliable  guesses. 

Strategic  Level  Thinking 

Strategic  thinking  is  a  deliberate  approach  to  thinking  about  a  situation  and  what  to  do.  It  involves 
thinking  broadly,  deeply,  and  into  the  future.  Broad — seeing/making  connections  across  the  organization 
and  outside  the  Army.  Deep — deeply  questioning  problems,  their  causes,  opportunities  to  improve,  and 
solutions.  Future — shaping  solutions  far  into  the  future  so  the  organization  can  implement  effective, 
lasting  change.  Involves  thinking  about  the  complex  and  dynamic  factors  that  go  well  beyond  the  typical 
and  familiar  situations. 

Cognitive  Flexibility/Mental  Agility 

Models  a  flexible  mindset,  willing  to  be  flexible  in  approach,  anticipates,  and  scans  for  changing 
conditions,  able  to  apply  fresh,  different  perspectives  to  problems. 

ADVISING 

Advise/Influence 

Advises  the  senior  leaders  and  command  teams  on  enlisted  and  noncommissioned  areas.  Uses 
appropriate  methods  of  influence  to  energize  others,  ranging  from  compliance  to  commitment  (pressure, 
legitimate  requests,  exchange,  personal  appeals,  collaboration,  rational  persuasion,  apprising, 
inspiration,  participation,  and  relationship  building). 

Communication 

Communicates  effectively  by  clearly  expressing  ideas  and  actively  listening  to  others. 


Nominative  CSM  Behaviors 

Given  that  the  final  nominative  CSM  competency  model  was  to  have  an  applied  use  in  directing  the 
scoring  of  the  exercises  and  simulations,  the  model  was  further  expanded  to  create  behaviorally 
anchored  rating  scales  (BARS)  for  each  competency.  Detailed  descriptions  of  three  effectiveness  levels 
were  created  for  each  behavior:  Opportunity  for  Improvement,  Solid,  and  Strength.  See  Table  2  for  a 
sample  of  the  nominative  CSM  competency  model  BARS  used  in  the  LDAC. 


Table  2:  Sample  LDAC  Behaviorally  Anchored  Rating  Scales 


Judgment/Decisionmaking 

Behavior 

Opportunity  for  Improvement 
{  —  or  A} 

Solid  {  0  or  ✓  } 

Strong  /  Strength  {  +  } 

Applies  the  critical 
thinking  needed  to 
identify  faulty  logic 
and  solution  pitfalls. 

Misses  opportunities  to  detect 
inconsistencies  that  could  lead 
to  questionable  conclusions. 

Appropriately  uses 
common  sense  in 
making  judgments. 

Consistently  shows  well- 
reasoned  judgment  that  is  based 
on  appropriate  logic  and  accurate 
identification  of  faulty  reasoning. 

Recognizes  the 
need  to  gain 
additional 
information. 

Focuses  on  readily  available 
information  or  uses  information 
that  has  minor  value,  when 
trying  to  understand  problems 
or  situations. 

Gathers  and  uses  the 
most  relevant 
information  needed  to 
understand  problems 
or  situations. 

Consistently  recognizes  and 
gathers  the  most  relevant 
information  needed  to  fully 
understand  problems  or 
situations. 
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Scenario  and  Center  Development 

LDAC  Activities  Overview 

As  stated  above,  a  key  component  of  the  assessment  center  methodology  is  based  on  using  multiple 
measures  (i.e.,  simulations,  tests,  interviews,  etc.)  with  a  standardized  and  objective  process.  The 
integrated  results  of  these  different  measures  then  provides  an  assessment  of  what  leaders  can  do  and 
their  potential  to  be  successful  in  senior  strategic  roles.  LDAC  activities  were  divided  into  three  segments, 
which  are  described  in  greater  detail  in  the  following  pages. 

Segment  1:  Orientation  and  Testing.  Personality  and  dispositional  measures  as  well  as  cognitive 
ability  tests  were  utilized  to  understand  the  leader’s  learning  approach,  work  style,  problem¬ 
solving  skills,  and  interpersonal  skills.  Information  from  these  tests  provided  insights  into  how  the 
leader  processed  information  and  approached  interpersonal  situations. 

Segment  2:  Simulations.  The  second  phase  began  with  an  in-depth  interview  with  a  lead 
assessor  and  provided  the  opportunity  to  understand  the  participant’s  work  experience, 
achievements,  and  skills,  as  well  as  career  goals  and  aspirations.  The  assessment  process 
continued  with  several  work-based  exercises  that  were  designed  to  simulate  challenges 
typically  faced  at  the  nominative  CSM  level. 

Segment  3:  Integrated  Feedback.  The  assessment  center  concluded  with  providing 
comprehensive  feedback  to  each  individual  participant  on  all  of  the  assessment  center 
components  (i.e.  simulations,  interview,  and  inventories).  The  participant’s  lead  assessor  worked 
with  him  or  her  to  understand  and  prioritize  this  feedback  in  relation  to  current  career  goals  and 
objectives.  Each  participant  received  a  written  report  that  integrated  all  of  the  information 
discussed  in  the  Integrated  Feedback  session. 

Test  Selection 

The  LDAC  utilized  cognitive  ability,  personality,  and  dispositional  measures,  which  were  integrated  with 
other  data  (e.g.,  interviews  and  simulation  scores)  from  the  center.  A  number  of  personality  and  cognitive 
ability  tests  were  reviewed  for  potential  inclusion  from  both  within  the  Army  as  well  as  from  external 
sources.  Given  the  unique  military  nature  of  the  mission  and  culture  of  the  Army,  emphasis  was  placed  on 
finding  tests  that  were  developed  within  the  Army  or  had  Army-specific  norm  comparison  groups. 

The  Army’s  Tailored  Adaptive  Personality  Assessment  System  (TAPAS)  was  considered  as  a  potential 
personality  assessment.  TAPAS  was  developed  by  the  Army  Research  Institute  (ARI)  using  Army 
populations  (Chernyshenko,  Stark,  &  Drasgow,  2010;  Drasgow,  2012).  The  TAPAS  was  developed  to 
assess  up  to  21  sub-dimensions  of  the  Big  Five  personality  factors  and  several  additional  personality 
characteristics  relevant  to  military  settings.  It  has  been  used  for  initial  military  accessions  and  placement 
(Knapp  &  Heffner,  2010;  Nye  et.  al,  2012).  The  TAPAS  has  an  adaptive  administration  method  that 
decreases  the  time  required  from  the  participant  and  also  has  Army-specific  norm  comparison  groups. 
Although  the  TAPAS  has  Army-specific  norms,  the  comparison  data  for  NCO  populations  were  at  rank 
levels  far  below  the  CSM  rank,  which  would  have  made  it  difficult  to  draw  comparisons  to  higher  levels  of 
leadership.  Secondly,  the  TAPAS  is  primarily  a  research-based  measure  and  has  not  yet  been  adapted 
for  individual  interpretation  or  use.  At  the  time  of  the  LDAC  administration,  ARI  did  not  have  an  individual 
feedback  report  designed  for  participants,  and  given  the  time  constraints  of  the  LDAC  project,  this  was 
not  something  that  could  be  created  prior  to  the  execution  of  the  center.  Finally,  the  constructs  assessed 
by  the  TAPAS  were  not  well  aligned  with  Army  leadership  doctrine  (e.g.,  ADRP  6-22)  nor  the  focal 
competencies  and  behaviors  being  assessed  via  the  LDAC.  For  these  reasons,  the  decision  was  made 
not  to  use  the  TAPAS  for  the  current  administration  of  the  LDAC. 

Two  external  and  two  internally  developed  measures  of  both  cognitive  ability  and  personality  were 
selected  for  nominative  LDAC.  Cognitive  ability  was  measured  using  two  assessments,  the  Army  Critical 
Thinking  Test  (ACTT)  and  the  Ravens  Standard  Progressive  Matrices  (SPM).  The  ACTT  was  developed 
and  validated  by  CAL  and  is  a  measure  of  critical  thinking  (Curnow,  Parish,  &  Fallesen,  2008).  The 
Ravens  SPM  is  an  abstract  reasoning  test  that  has  established  validity  and  reliability  (Raven,  Raven,  & 
Court,  2000)  and  is  widely  used  in  the  private  sector  for  leader  development  initiatives  (Ree  &  Carreta, 
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2002,  Gonzalez,  Thomas,  &  Vanyukov,  2005).  The  use  of  these  two  cognitive  ability  tests  provided  an 
assessment  of  an  individual’s  reasoning  capability  and  potential,  as  they  each  captured  different 
components  of  general  cognitive  ability.  Two  aspects  of  personality  and  disposition  were  also  assessed. 
The  Deep  Learning  Orientation  Battery  (DLOB),  developed  by  CAL,  provides  dispositional  indicators 
related  to  the  Intellect  attributes  (see  ADRP  6-22)  and  assesses  the  individual’s  use  of  cognitive  and 
meta-cognitive  strategies.  Interpersonal  personality  factors  and  work  style  constructs  related  to  leadership 
were  assessed  using  the  Workplace  Personality  Inventory  II  (WPI-II),  which  has  established  validity  and 
reliability  and  is  widely  used  in  the  private  sector  for  leader  development  initiatives  (Pearson,  201 3). 
Confidential  individualized  feedback  reports  on  all  tests  were  provided  to  CSM  participants  and  no  one 
else.  See  Appendix  B  for  details  on  measures  administered  as  a  part  of  the  LDAC. 

Test  Scoring  Protocol 

The  raw  percentile  scores  for  the  two  externally  validated  tests,  the  WPI-II  and  the  Raven’s  SPM,  were 
transformed  into  a  common  number  scale  and  integrated  into  a  corresponding  competency  score.  LDAC 
developers  decided  not  to  integrate  the  CAL-developed  dispositional  and  cognitive  ability  measures  (e.g. 
the  DLOB  and  the  ACTT),  as  there  was  insufficient  validity  evidence  to  support  their  integration  into  the 
overall  assessment  scores.  A  test  conversion  calculator  was  used  to  transform  the  raw  percentile  scores 
into  1-5  rating  scores  that  could  be  numerically  aggregated  with  the  other  LDAC  exercise  and  simulation 
scores  (see  Appendix  C  for  detailed  description  of  influence  test  scoring).  The  day  after  completion  of 
assessments,  participants  were  provided  reports  of  their  results  on  each  individual  test  (e.g.,  WPI, 

Raven’s  SPM,  ACTT,  DLOB)  during  a  feedback  session  with  their  lead  assessor.  Each  lead  assessor 
helped  the  leader  interpret  and  integrate  the  results  with  their  broader  feedback  themes  received  during 
the  LDAC. 

Influence  Exercise 

Participants  were  given  one  hour  to  complete  a  six  part  influence  exercise  that  included  a  variety  of 
different  questions  and  situational  judgment  tests  to  assess  understanding  of  influence  techniques  and 
their  application  as  outlined  in  Army  leadership  doctrine  (ADRP  6-22).  The  influence  exercise  was  drawn 
from  three  sets  of  materials  that  were  developed  by  CAL  and  ARI.  One  set  of  CAL  materials  was 
developed  to  evaluate  instruction  on  influence  for  Army  leaders  who  were  training  to  become  foreign 
security  force  (FSF)  advisors.  The  training  material  was  based  on  instruction  developed  in  2008  on 
commitment,  compliance,  and  resistance  for  FM  6-22  (Department  of  the  Army,  2006).  The  other  set  of 
CAL  items  was  selected  from  situational  judgment  tests  that  were  developed  to  assess  leadership 
competencies.  The  ARI  material  consisted  of  items  from  a  self-assessment  tool  that  used  situational 
judgment  test  items  and  self-reflection  on  preferred  influence  strategies  (Zbylut,  Wisecarver,  Foldes,  & 
Schneider,  2010).  All  materials  were  screened  to  select  items  applicable  to  non-commissioned  officers 
and  for  nominative-level  CSMs.  The  items  also  assessed  participant’s  ability  to  recognize  and 
appropriately  address  different  types  of  resistance  in  others.  These  items  were  drawn  from  past  work  by 
CAL  and  ARI  on  influence  techniques  (see  Appendix  C  for  more  detail).  Scores  for  the  influence  exercise 
were  converted  to  a  common  1  to  5  metric  used  in  the  other  nominative-CSM  exercises  (see  Appendix  C 
for  detailed  description  of  influence  test  scoring).  An  influence  exercise  interpretation  guide  was  provided 
to  each  lead  assessor  to  assist  with  interpreting  the  influence  scores  and  to  provide  guidance  on  how  to 
provide  verbal  and  written  feedback  to  their  participants  on  the  results  during  the  overall  feedback 
session. 

Overall  Scenario  Development  Process 

The  development  of  the  LDAC  scenario,  which  would  provide  the  background  and  story  foundation  for  the 
simulations,  was  developed  through  a  three-stage  approach.  This  approach  included  soliciting  guidance 
from  experts  within  the  Army  currently  utilizing  an  assessment  center  methodology,  as  well  as  reviews 
with  SMEs.  The  key  aspects  of  the  scenario  development  are  described  in  the  next  section  and  Figure  2 
presents  a  summary  of  the  actions  within  each  stage. 
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Figure  2:  Overall  LDAC  Scenario  Development  Process 
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Learning  from  Existing  Army  Assessment  Centers 

Given  that  the  assessment  center  methodology  is  currently  in  use  in  different  segments  of  the  Army,  CAL 
researchers  set  out  to  gain  an  understanding  of  these  administrations  and  their  best  practices.  In 
February  2014,  CAL  researchers  met  with  representatives  from  the  75th  Ranger  Regiment  to  learn  what 
tests  they  use  as  part  of  the  selection  phase  of  their  assessment  center.  In  March  2014,  CAL  researchers 
visited  the  Asymmetric  Warfare  Group  (AWG)  to  observe  and  leverage  their  expertise  in  administering 
assessment  centers  to  assess  and  select  candidates  for  their  AWG  cadre.  The  researchers  observed  a 
week-long  set  of  exercises  that  followed  standard  assessment  center  practices.  The  AWG  used  a 
threaded  scenario  to  evoke  behaviors  associated  with  specific  competencies  and  attributes  identified  for 
operational  advisors  (e.g.,  self-confidence,  temperament,  perspective-taking,  problem  solving,  conceptual 
capacity,  and  communications).  Candidates  were  also  given  standardized  personality  inventories  and 
cognitive  ability  tests,  though  none  of  these  utilized  Army  norms.  Candidates  did  not  receive  feedback  on 
their  assessment  results  during  or  after  their  participation.  The  candidates  who  were  selected  from  the 
assessment  center  process  would  enter  another  phase  of  training  and  assessment.  It  was  at  that  point 
when  the  leader  received  feedback,  including  results  from  their  standardized  testing.  The  AWG 
assessment  center  was  initially  developed  in  2006  and  relies  upon  a  consistent  and  well-trained  staff  of 
role  players  and  psychologists  to  provide  reliable  and  calibrated  ratings.  The  AWG  assessment  center 
has  also  been  evaluated  with  respect  to  its  criterion  validity,  or  how  well  it  predicts  success  of  candidates 
in  the  advisor/operator  role.  CAL  researchers  obtained  information  on  the  development  of  the 
assessments,  administration  and  operations,  use  of  the  evaluations  in  their  boards  for  selecting  AWG 
cadre,  and  the  validation  of  the  assessments. 

Scenario  and  Simulations  Creation 

Leveraging  the  lessons  learned  from  other  Army  assessment  centers,  the  scenario  development  for 
LDAC  centered  on  concepts  that  included  multiple  CSM  competencies,  was  capable  of  being  executed  by 
a  limited  staff  of  role  players  and  assessors,  sufficiently  detailed  to  serve  as  the  foundation  for 
simulations,  and  realistic  enough  to  have  face  validity  with  participants.  The  key  intent  was  to  identify  an 
overarching  scenario  that  required  individuals  to  synthesize  large  amounts  of  information  and  guidance 
from  a  GO,  and  then  make  decisions  and  communicate  this  information  to  others.  SMEs  (retired  CSMs) 
were  engaged  to  help  identify  and  develop  the  overarching  scenario,  which  included  a  multi-stage  role- 
play  simulation.  As  part  of  the  scenario  development,  all  information  was  provided  to  additional  SMEs  for 
review  and  recommendations.  These  SMEs  possessed  expertise  with  brigade  command,  Judge 
Advocate  General  processes  and  regulations,  or  recent  forward  operations.  A  pilot  rehearsal  of  the 
scenario  and  simulations  was  then  conducted  with  observers  from  CAL’s  team  of  research  psychologists 
and  observers  from  the  School  for  Command  Preparation.  After  action  reviews  (AARs)  were  then 
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conducted  and  revisions  were  made  to  both  the  content  and  the  sequencing  of  the  material.  Role-play 
simulations  required  the  CSM  participants  to  quickly  assimilate  and  integrate  information,  communicate 
the  commander’s  intent,  and  exercise  appropriate  methods  to  influence  key  stakeholders.  Participants 
were  provided  read-ahead  materials  prior  to  their  arrival  at  the  LDAC,  as  well  as  immediately  prior  to  each 
simulation  in  order  to  prepare  for  the  upcoming  challenge. 

LDAC  Component  Competency  Coverage 

Each  activity,  exercise,  and  test  utilized  in  the  LDAC  was  designed  to  capture  a  specific  target  factor  and 
competency  identified  in  the  initial  model  development.  Based  upon  expert  understanding  of  the  target 
dimensions  and  exercises,  LDAC  developers  determined  the  competencies  that  should  be  elicited  by  the 
exercises  (see  Table  3). 


Table  3:  Exercise  by  Competency  Mapping 


Competency 

Role  Playing  Simulations 

Testing 

Influence 

Exercise 

Structured 

Interview 

Written 

Memo 

CG 

Debrief 

Congressional 

Staffer 

Interaction 

Judgment/ 

Decisionmaking 

X 

X 

X 

X 

X 

X 

Strategic  Level 
Thinking 

X 

X 

X 

X 

X 

Cognitive 

Flexibility/  Mental 
Agility 

X 

X 

X 

X 

Advise/Influence 

X 

X 

X 

X 

X 

X 

Communication 

X 

X 

X 

X 

X 

Validation 

The  validity  of  the  assessment  center  methodology  relies  upon  content  validity  (the  identification  of 
dimensions  that  are  relevant  to  the  job  in  question),  and  construct  validity  (that  the  assessments  measure 
what  they  are  intended  to  measure).  As  previously  stated,  content  validity  was  established  through  a  mix 
of  rational  and  empirical  job  modeling  methods.  The  resulting  LDAC  dimensions  were  based  on  available 
survey  research  and  organizational  literature.  Specifically,  dimensions  understood  to  be  ‘developable’ 
were  targeted.  In  this  case  the  dimensions  were  distilled  from  three  primary  sources  and  reviewed  by 
SMEs  described  in  detail  above. 

For  construct  validity,  assessments  were  chosen  that  had  pre-existing  evidence  of  convergent  and 
divergent  validity  for  the  targeted  dimensions,  in  the  form  of  moderate  to  large  correlations  with  relevant 
external  variables.  For  dimensions  in  which  no  assessments  were  available,  CAL  developers  created 
tools  using  best  practices  for  assessment  design.  The  LDAC  was  not  intended  to  provide  information 
relevant  to  selection  or  promotion  as  criterion  validity  evidence  of  the  assessment  center  had  not  yet  been 
established.  In  other  words,  the  assessments  were  not  selected  or  developed  based  on  their  ability  to 
predict  performance  as  a  Nominative  CSM.  Further  validation  work  is  needed  to  use  the  specific  set  of 
assessments  for  the  purposes  of  selection  and/or  promotion.  Likewise,  this  demonstration  did  not  validate 
the  use  of  the  LDAC  as  a  leadership  development  technique,  which  would  require  data  on  the 
performance  of  nominative  CSMs  (those  who  attended  the  LDAC  and  those  who  did  not).  These  data 
could  be  gathered  in  the  future.  However,  research  does  indicate  that  participation  in  assessment  centers 
can  contribute  to  improved  short-term  managerial  performance  (Engelbrecht  &  Fischer,  1995)  and 
participants  are  more  likely  to  be  active  in  working  their  individual  development  plans  a  year  later  (Byham, 
2005).  Assessment  centers  also  provide  realistic  job  previews,  arguably  decrease  attrition,  and  increase 
job  satisfaction.  Multiple  studies  have  shown  the  validity  of  assessment  centers  for  predicting  supervisor 
performance  to  range  from  .28  (Hermelin,  Lievens,  &  Robertson,  2007)  to  .36  (Gaugler,  Rosenthal, 
Thornton  &  Bentson,  1987),  with  some  studies  approaching  corrected  validity  coefficients  of  .70. 
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Staff  Training  and  Calibration 


LDAC  Demonstration  Staff 

The  LDAC  demonstration  utilized  a  total  of  14  staff  taken  from  available  and  qualified  manpower  within 
CAL  and  MCCoE.  These  included  five  assessors,  seven  role  players  with  previous  military  experience 
(two  of  the  seven  were  designated  substitutes  if  back  up  was  needed),  one  report  writer/editor,  and  one 
senior  NCO  who  served  as  the  LDAC  coordinator.  Two  assessors  also  served  dual  roles  as  LDAC 
program  director  and  lead  administrator.  To  further  protect  the  confidentiality  of  participants  and  results 
from  unauthorized  release,  all  LDAC  staff  were  required  to  sign  non-disclosure  agreements  with  regards 
to  who  participated  and  what  was  observed.  In  keeping  with  best  practices  outlined  by  Guion  (1998) 
assessors  were  behavioral  psychologists  who  had  undergone  training  (see  below)  to  ensure  that  they 
understood  the  performance  dimensions  to  be  assessed,  and  were  competent  to  produce  accurate  and 
reliable  oral/written  performance  information.  Research  has  provided  support  for  the  use  of  trained 
behavioral  psychologists  as  assessors,  finding  their  usage  associated  with  higher  validities  than  when 
managers  or  organizational  leaders  are  used  as  assessors  (Gaugler  et  al.,  1987). 

Assessor  Training  and  Calibration 

Consistent  with  assessment  center  research  and  best  practices  (International  Task  Force  2000;  Thornton 
&  Zorich,  1980;  Thornton  &  Rupp,  2006),  all  assessors  were  required  to  attend  two  six-hour  blocks  of 
training  and  calibration  in  preparation  for  the  execution  phase  of  LDAC.  During  the  first  block  of  training, 
assessors  received  instruction  on  their  dual  responsibility  within  the  LDAC,  as  both  assessor  and  coach. 
This  dual  responsibility  has  been  demonstrated  as  a  more  cognitively  taxing  administration  method  that 
requires  more  sophisticated  and  experienced  staff  (Atchley,  Smith,  &  Hoffman,  2003).  With  respect  to 
their  evaluation  duties,  assessors  were  trained  per  best  practices  for  behavioral  observation  and  optimal 
categorization/scoring  of  observed  stimuli  (Gorman  &  Rentsch,  2009;  Lievens,  2001;  Roch  &  O’Sullivan, 
2003;  Schleicher,  Day,  Mayes,  &  Riggio,  2002;  Spychalski,  Quinones,  Gaugler,  &  Pohley,  1997;  Woehr, 

1994) .  Simulation  role  plays  during  the  LDAC  were  not  video  recorded  due  to  time  and  resource 
constraints.  Although  recording  the  simulations  might  have  been  useful  for  training  purposes  in  future 
iterations  of  the  LDAC,  research  suggests  that  making  video  recordings  available  to  assessors  increases 
the  assessment  time  per  participant  without  producing  a  meaningful  increase  in  accuracy  (Ryan  et  al., 

1995) .  Assessors  were  also  trained  on  best  methods  for  establishing  an  effective  coaching  relationship 
with  the  participant  to  aid  in  translating  the  feedback  into  realistic  developmental  guidance. 

The  second  block  of  training  focused  on  calibrating  the  assessors  to  the  behaviors  elicited  in  the 
simulations  and  the  optimal  interpretation  of  the  BARS  for  each  competency.  Assessors  watched  and 
independently  scored  practice  videos  of  both  simulations.  Assessors  then  came  together  as  a  group  to 
discuss  and  calibrate  score  assignment. 

Role  Player  Training 

Role  players  were  required  to  attend  a  six-hour  block  of  training  prior  to  the  LDAC,  during  which  they 
were  instructed  on  LDAC  methodology,  the  nominative  CSM  competency  model,  as  well  as 
responsibilities  as  role  players.  Role  players  were  CAC  Army  civilians  who  had  retired  from  service  as 
field  grade  officers.  Given  resource  constraints  for  the  demonstration,  role  players  learned  to  play  two 
roles,  a  GO  for  the  CG  Debrief  simulation  and  a  staffer  for  the  Congressional  Staffer  Engagement 
simulation.  Role  players  were  provided  scripts  for  each  simulation  that  included  detailed  prompts  or 
predetermined  statements.  These  were  used  consistently  across  all  role  players  to  ensure  participants 
were  given  similar  opportunities  to  react  to  simulation  stimuli.  Research  has  noted  the  benefits  of 
providing  detailed  scripts  and  specific  prompts  for  role  players,  which  can  significantly  enhance  the 
construct  validity  of  assessment  centers  (Schollaert  &  Lievens,  201 1 ;  Schollaert  &  Lievens,  201 2). 

Role  players  were  encouraged  to  play  the  role  objectively  and  consistently  to  ensure  all  participants  were 
exposed  to  the  same  character  portrayal  and  stimuli.  Role  players  were  also  trained  on  the  use  of  the 
Role  Player  Observation  form  to  record  their  perceptions  of  the  participant  (see  scoring  section  below  for 
more  detail).  The  day-long  training  concluded  with  a  guided  practice  session,  which  allowed  LDAC  staff  to 
calibrate  each  role  player  to  the  scenario  details  and  ensure  consistent  depiction  of  each  character  and 
role. 
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LDAC  Administration  and  Execution 


Schedule 

Participants  were  contacted  three  weeks  prior  to  the  onsite  portion  of  the  LDAC  with  a  brief  orientation  to 
the  assessment  center  experience  and  instructions  to  complete  the  two  online  personality  tests  (WPI-II 
and  DLOB).  The  main  portion  of  the  LDAC  demonstration  was  conducted  from  17-23  October  2014  in 
conjunction  with  on-going  activities  for  the  Command  Sergeants  Major  Developmental  Program  (CSMDP) 
held  at  Fort  Leavenworth,  KS.  The  LDAC  activities  occurred  either  before  or  after  instruction  in  CSMDP 
classrooms  and  did  not  interfere  with  the  standard  curriculum.  The  LDAC  was  officially  launched  with  a 
kick-off  working  lunch  briefing  from  the  Combined  Arms  Center  (CAC)  Commanding  General  and  CSM. 
Participants  were  then  given  a  specific  LDAC  overview  brief  from  the  lead  administrator,  which  was 
followed  by  the  proctored  administration  of  both  cognitive  ability  tests  (e.g.  Ravens  SPM  and  ACTT). 
Consistent  with  best  practices  research  on  assessment  centers  (Kleinmann,  Kuptsch,  &  Koller,  1996; 
Kolk,  Born,  &  van  der  Flier,  2003),  participants  were  also  provided  with  a  detailed  description  of  the 
nominative  CSM  competency  model  and  the  behaviors  that  would  be  assessed  during  the  LDAC. 

The  Interview  was  the  first  and  longest  interactive  exercise  (e.g.  60  minutes)  completed  by  participants 
and  was  led  and  scored  by  the  lead  assessors.  The  interview  was  followed  by  the  Influence  Exercise, 
which  was  proctored  by  the  LDAC  coordinator.  Participants  were  then  given  30  minutes  to  prepare  for  the 
simulations,  each  of  which  was  15  minutes  in  duration.  The  interview  and  all  simulations  were  conducted 
in  the  same  room  for  the  participant  (e.g.  the  participant’s  “office”).  Due  to  schedule  constraints, 
participants  had  different  role  players  but  the  same  assessor  across  both  simulations  (e.g.  both  the  CG 
Debrief  and  the  Congressional  Staffer  engagement).  See  Table  4  for  a  breakdown  of  time  allotment  for 
LDAC  activities  and  Figure  3  for  the  flow  of  the  simulation  segment. 


Table  4:  Simulation  Duration 


Simulation/Exercise 

Time  Allotted 

Assessor 

Role  Player 

Interview 

60  min 

Each  Lead 

n/a 

Written  Influence 

Exercise 

60  min 

Scorer 

n/a 

CG  Debrief 

15  min 

A 

1 

Written  Memo 

15  min 

B 

n/a 

Congressional  Staffer 
Engagement 

15  min 

A 

2 

10 


Figure  3:  Simulation  Flow 


15  min  ,  15  min  .  15  min  ,  15  min 


Participants  were  separated  into  three  cohorts,  each  consisting  of  4-5  participants,  with  staggered 
administration  times  throughout  the  week.  Each  assessor  was  assigned  to  be  the  lead  assessor  for  2-3 
participants,  up  to  one  participant  from  each  cohort  (maximum  1 :3  assessor  to  participant  ratio).  All 
cohorts  completed  their  simulations,  interviews,  and  exercises  on  the  first  two  days  of  the  assessment 
center  with  his  or  her  integrated  feedback  with  their  lead  assessor  occurring  one  full  day  later.  The  total 
time  commitment  for  the  LDAC  was  roughly  6.75  hours  per  participant. 

Simulation  Scoring 

As  previously  discussed,  behaviorally  anchored  rating  scales  (BARS)  were  developed  based  on  the 
targeted  competencies  for  each  of  the  four  exercise  assessment  points  (e.g.  interview,  CG  Debrief, 
Congressional  Staffer  Engagement,  and  Written  Memo).  As  per  assessment  center  best  practices,  a  post¬ 
exercise  dimension  rating  method  was  used  for  each  simulation,  where  assessors  evaluated  participants 
on  each  competency  using  a  unique  set  of  BARS  that  best  reflected  the  behaviors  elicited  by  the 
requirements  of  the  exercise  (e.g.,  Robie,  Adams,  Osburn,  Morris,  &  Etchegaray,  2000;  Sackett  & 
Tuzinski,  2001).  Each  behavior  was  scored  on  a  5-point  response  scale  that  included  three  behavioral 
anchors  (e.g.,  1 -needs  development,  3-solid,  5-strength).  During  the  LDAC  training  and  calibration, 
assessors  were  instructed  to  evaluate  participants  based  on  the  actual  behaviors  observed  and  not  on 
perceived  intentions  or  motivations.  The  simulation  score  sheet  (see  Figure  4)  auto-generated  the 
statistical  average  for  each  competency  based  on  the  ratings  assigned  for  each  behavior.  The  final 
competency  score  was  determined  based  on  the  assessor’s  judgment  and  was  rounded  to  the  whole  or 
half  number.  To  enhance  the  quantitative  rating,  assessors  also  provided  qualitative  comments  to 
describe  the  participant’s  behavior  relative  to  each  competency. 
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Figure  4:  Sample  Simulation  BARS  Score  Sheet 


Simulation:  Congressional  Staffer  Engagement 

Participant:  John  Appleseed 
Assessor: 

Judgment  /  Decision  Making 


Competency  Model 
Behavior 

1  2  3  4  5 

Behavior 

SCORE 

Engages  in  thoughtful 
assessment:  demonstrates 
sound  judgment. 

Misses  opportunities  to  sufficiently 
consider  important  factors  when 
evaluating  situations,  leading  to 
incomplete  or  superficial  judgments. 

Demonstrates  thoughtful  analyses  and 
sound  judgment  on  issues. 

Consistently  makes  sound  judgment  on 
issues,  appropriately  balancing  thoughtful 
analyses  with  decisiveness. 

Relates  and  compares 
information  from  different 
sources  to  identify  possible 
cause-and -effect 
relationships. 

Considers  situations  at  face  value, 
missing  opportunities  to  look  past 
symptoms  and  probe  for  underlying 
causes 

Sufficiently  probes  into  situations  to 
integrate  information  from  different 
sources  and  determine  possible  root 
causes 

Probes  deeply  past  symptoms  to  integrate 
information  from  different  sources  and 
determine  possible  root  causes. 

Written  Notes:  (type  notes  in  cell  below) 

Num.  Avg. 

Score  for  Judgment 

ItDIV/Of 

□ 

'If  behavior  is  'Not  observed"  then  do  not  place  a  rating  in  the  box 


N/O 


Assessors  were  given  30  minutes  to  complete  the  BARS  scoring  guide.  Following  the  interactive 
simulations  (e.g.  CG  Debrief  as  well  as  the  Congressional  Staffer  engagement),  assessors  and  role 
players  also  discussed  the  performance  of  the  participants.  Role  players  were  encouraged  to  complete 
observation  sheets  to  record  their  perception  of  the  participant  following  the  discussion  with  the  assessor. 
This  Role  Player  Observation  Form  was  intended  as  a  way  to  help  role  players  organize  their  perceptions 
and  capture  general  insights  from  the  simulation.  When  completed,  role  player  observation  forms  were 
attached  to  the  assessor’s  BARS  scoring  guide  but  did  not  formally  affect  the  statistical  rating  for  each 
competency. 

Scoring  Integration 

Upon  completion  of  the  exercises  and  simulations,  all  available  assessment  data  on  each  participant  was 
compiled  for  the  respective  lead  assessors.  The  participant’s  file  included  test  reports,  interview  score 
sheets,  simulation  score  sheets,  role  player  observation  forms,  and  an  influence  exercise  report.  Lead 
assessors  were  also  provided  with  an  Integration  Grid,  which  represented  a  statistical  combination  of  all 
scores  across  all  competencies  (see  Figure  5).  This  integration  grid  was  the  compilation  of  multiple 
ratings  made  by  multiple  assessors  evaluating  multiple  exercises.  Assessors  were  allotted  three  hours  for 
integration  where  all  available  assessment  data  on  the  participant  were  integrated  together  to  identify  key 
themes,  patterns,  and  behavioral  triggers.  Although  a  statistical  aggregate  of  all  ratings  for  each 
competency  was  provided  on  the  integration  grid,  assessors  were  also  required  to  make  the  final  rating 
determination  for  each  competency.  Similar  to  the  simulation  score  sheets,  the  final  competency  rating 
was  on  a  5-point  scale  (e.g.,  1 -needs  development,  3-solid,  5-strength).  For  the  final  competency  rating,  a 
hybrid  method  was  utilized  that  relied  upon  both  quantitative  and  qualitative  methods.  This  process  is 
recommended  by  industry  best  practices  when  using  expert  assessors  (Thornton  &  Rupp,  2006).  The 
hybrid  method  was  advantageous  as  it  allowed  for  the  benefits  of  both  quantitative  and  qualitative 
integration  methods  to  be  utilized.  A  hybrid  method  included  the  benefit  of  having  a  statistical  aggregation 
of  scores,  which  has  been  supported  by  research  with  the  qualitative  method  that  has  greater  face  validity 
for  being  objective  yet  also  leveraging  the  expertise  of  the  assessor.  As  the  purpose  of  the  LDAC  was 
developmental  and  not  administrative,  no  overall  assessment  rating  was  calculated.  Following  the 
integration  period,  assessors  held  a  review  meeting  which  provided  those  involved  with  the  participant  to 
discuss  his/her  performance  and  provide  clarification  from  the  BARS  simulation  score  sheets. 
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Figure  5:  Integration  Grid 


CSM  LEADER  DEVELOPMENT  ASSESSMENT  CENTER 
Integration  Grid 


Participant:  CSM  Chris  Gibbs 

Lead  Assessor:  Dr.  Assessor 


Cohort:  2 

Date:  22-Oct-14 


ICCEZBj 


Opportunities  to  Assess 


Competencies 

H 

1 

£ 

Interview 

Influence  Exercise 

Written  Memo 

CG  Debrief 

Staffer  Engagement 

OVERALL  Average 

Final  Average 

Assessor  initials 

ran 

raj 

|  JB 

ow 

INTELLECT/COGNITION 

284 

3 

Judgment/Decision  Making 

1.35 

3.0 

3.50 

2.0 

3.5 

3.5 

2.81 

3 

Strategic  Level  Thinking 

1.51 

2.5 

Izol 

3.5 

4.0 

2.70 

2.5 

Cognitive  Flexibility/  Mental 

Agility 

2.01 

3.5 

3.0 

3.5 

3.00 

3.5 

ADVISMG 

2.78 

3 

Advise/Influence 

2.30 

3.0  I  2.00  I  2.0 

3.0 

3.0 

2.55 

3 

Communication 

1.53 

3.0 

3.5 

3.01 

3.5 

LEGEND 


Rate 
Do  Not  Rate 
Numerical  Average 
Final  Rating 


Participant  Feedback 

Each  participant  received  IV2  hours  of  feedback  with  his  or  her  lead  assessor  two  days  after  completing 
the  simulation  phase  of  the  LDAC.  The  purpose  of  the  feedback  session  was  to  provide  integrated 
feedback  about  the  participant’s  strengths  and  developmental  areas  relative  to  the  nominative  CSM  role. 
While  it  may  have  been  discussed  in  relation  to  integrated  themes,  the  feedback  did  not  focus  on  the 
participant’s  performance  on  one  specific  simulation  or  exercise.  Rather,  the  behaviors  exhibited  during 
the  LDAC  were  used  to  illustrate  integrated  themes  and  points,  following  best  practices  for  impactful 
feedback  (Francis-Smythe  &  Smith,  1997;  London,  1997).  Participants  were  also  provided  test  reports 
(e.g.,  WPI-II,  Raven’s  SPM,  ACTT,  DLOB)  as  well  as  guidance  on  how  to  interpret  and  incorporate  them 
into  the  overall  feedback.  The  lead  assessor’s  role  was  to  put  the  feedback  in  context  and  provide 
guidance  for  the  participant  on  how  to  prioritize  this  feedback  in  relation  to  their  own  specific  career  goals, 
which  for  some  did  not  include  the  nominative  CSM  position.  As  such,  the  feedback  session  was  more 
interactive,  with  the  participant  playing  a  key  role  in  discussing  and  integrating  the  data  into  their 
subsequent  development.  A  layout  of  the  feedback  roughly  followed  the  method  recommended  by 
Thornton  and  Rupp  (2006),  which  included  an  introduction,  discussion  of  participant  goals,  discussion  of 
specific  test  results,  discussion  of  integrated  feedback,  development,  and  context,  and  a  conclusion. 
Given  research  on  feedback  acceptance,  assessors  were  encouraged  to  focus  on  the  developmental 
nature  of  the  experience  as  well  as  on  the  long-term  benefits  of  obtaining  such  feedback  at  this  time  in 
their  career  (Kluger  &  DeNisi,  1996).  The  importance  of  the  assessor’s  role  in  this  feedback  session 
cannot  be  understated,  as  it  has  a  large  impact  on  the  participant’s  perception  of  the  overall  experience, 
as  well  as  their  acceptance  of  the  feedback  itself  (Bell  &  Arthur,  2008;  Ryan,  Brutus,  &  Greguras,  2000). 
Feedback  acceptance  has  been  shown  to  be  a  critical  determinant  in  whether  an  individual  elects  to 
engage  in  developmental  activities  or  behavioral  change  following  a  leader  development  initiative 
(Ashford,  1986;  Brett  &  Atwater,  2001).  As  such,  it  was  essential  that  assessors  establish  the  appropriate 
environment  and  interaction  quality  with  their  participants  during  the  LDAC  process. 
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Participant  Reports 

Participants  were  sent  a  comprehensive  written  report  from  his  or  her  lead  assessor  approximately  two 
weeks  following  the  completion  of  the  LDAC.  These  reports  provided  participants  their  LDAC  feedback  in 
a  different  mode  than  the  face-to-face  session,  which  has  been  shown  to  increase  acceptance  and 
understanding  of  feedback  (Brutus,  2009).  As  with  the  results  from  the  LDAC,  these  reports  were 
confidential  and  sent  only  to  the  participant.  LDAC  reports  included  a  description  of  the  process  for  which 
participant  scores  on  the  assessment  exercises  were  generated  and  integrated  together.  Research  has 
shown  that  providing  such  procedural  detail  on  how  the  ratings  and  conclusions  were  generated  to 
participants  can  increase  feedback  acceptance  and  perceptions  of  credibility  (Fey,  Anseel,  &  Wille,  201 1 ). 
The  reports  also  included  a  detailed  description  of  the  integrated  themes  discussed  during  the  feedback 
session,  their  aggregate  scores  by  competency,  and  tailored  developmental  suggestions.  Providing 
detailed  and  behaviorally-specific  results  can  increase  individual  acceptance  and  facilitate  usage  of 
feedback  for  subsequent  developmental  actions  (Goodman  &  Wood,  2004).  The  full  integration  grid  and 
individual  test  results  were  not  included  in  the  participant’s  report. 

Data  Usage  and  Confidentiality 

Feedback  reports  and  results  from  the  LDAC  were  not  part  of  the  participant’s  formal  evaluation  process 
and  were  not  shared  with  his  or  her  chain  of  command.  All  results  were  kept  strictly  confidential  and 
were  not  released  outside  of  the  CAL  team  of  assessors.  CAL  was  the  only  agency  with  access  to  the 
full  set  of  raw  data  for  the  sole  purposes  of  continuing  development  and  validation  efforts.  In  accordance 
with  standards  set  forth  by  the  American  Psychological  Association  (2002),  assessment  data  were  not 
shared  with  any  third  parties  external  to  CAL  and  internal  access  was  limited  to  individuals  who  were 
appropriately  qualified  to  conduct  psychological  research  with  documented  training  and  experience  in 
interpreting  test  data  (e.g.  licensed  or  advanced  degree  in  psychology).  All  LDAC  staff  and  role  players 
were  required  to  sign  non-disclosure  agreements  with  regards  to  their  participation  and  what  they 
observed.  Assessment  data  are  maintained  securely  by  CAL  for  future  follow-up  with  participants.  The 
necessary  firewalls  and  online  safeguards  were  also  established  to  protect  the  online  portions  of  the 
LDAC  and  response  data  from  unauthorized  access. 
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LDAC  Findings 


Participant  Experience 

The  LDAC  provided  participants  with  a  structured  and  objective  assessment  of  their  capabilities  and 
potential  to  be  successful  in  higher-level  strategic  Army  roles.  Participants  received  feedback  and  worked 
individually  with  assessors  to  enhance  their  leadership  capabilities,  and  to  increase  their  self- 
awareness  and  confidence.  Participants  had  the  opportunity  to  provide  their  personal  reactions  to  the 
developmental  experience  upon  completion  of  all  LDAC  activities.  A  follow-up  survey  developed  by  CAL 
researchers  consisted  of  29  items  and  was  sent  to  all  participants  the  week  following  the  LDAC.  Eleven 
of  the  fourteen  participants  responded  (78%  response  rate).  Overall,  participants  expressed  satisfaction 
with  the  developmental  experience  and  thought  the  time  with  the  lead  assessor  was  valuable  for  their 
development.  When  reflecting  on  the  LDAC  experience,  one  participant  stated,  “ This  is  the  best 
counseling  I  have  received  in  27  years  of  service.  Our  NCO  corps  needs  this  type  of  evaluation.  I  want 
my  General  Officers  to  have  confidence  in  their  CSMs.”  A  summary  of  the  post  survey  results  is  shown 
below  in  Figure  6. 

Figure  6:  Summary  Post-LDAC  Survey  Results 


Overall  how  satisfied  were  you  wth  the  LDAC  experience 


LDAC  process  was  clealy  explained 


Assessor  was  effectwe  at  building  trust 


Trusted  results  will  be  kept  confitfcnti^ 


LDAC  is  worth  the  time 


I  am  more  motivated  to  develop  leadership  after  completing  LDAC 


Feedback  was  accurate 


LDAC  provided  insist  into  m.  capabilities 


LDAC  was  beneficial  to  my  development  as  a  leader 


□  Number  Responding  Favorably  out  of  11 


In  order  to  assess  the  longer-term  impact  of  the  LDAC  experience  on  development,  participants  were 
contacted  by  their  lead  assessor  six  months  following  the  close  of  the  LDAC.  A  structured  set  of 
questions  was  developed,  and  participants  were  given  the  option  of  discussing  the  questions  via  phone 
with  their  lead  assessor  or  providing  their  written  responses.  Follow-up  responses  were  received  from 
six  participants.  Consistent  with  the  feedback  received  immediately  following  the  LDAC,  participants 
valued  the  insight  and  interaction  gained  from  working  with  the  assessor  and  the  unbiased  approach  to 
assessing  their  skills.  One  participant  responded,  “/  appreciated  the  assessor’s  candor  and  clarity  of 
explanations  at  the  feedback  meeting.  Given  the  fact  that  the  assessor  “did  not  know  me  from  Adam,”  I 
considered  his  feedback  as  unbiased  and,  as  such,  valuable  to  my  self-development."  Overall, 
participants  who  responded  also  stated  they  had  been  motivated  following  the  LDAC  to  spend  time 
working  on  their  development  as  a  leader,  on  weaknesses  identified  during  the  LDAC  process,  or 
ensuring  others  who  they  supervise  or  mentor  are  more  informed  and  involved  in  their  development 
process.  Given  the  developmental  nature  of  the  LDAC,  no  performance-related  data  were  collected  as  a 
part  of  the  follow-up  survey,  though  this  kind  of  information  would  be  necessary  for  validation  of  the 
LDAC  if  it  is  to  be  used  for  future  administrations. 
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Return  on  Investment 

The  development,  validation,  and  administration  of  the  LDAC  demonstration  required  a  significant 
amount  of  resources  and  time  from  CAL  and  other  organizations.  As  can  be  seen  in  Figure  7,  the  largest 
contributing  source  for  development  and  administration  was  the  team  of  CAL  research  psychologists. 
One  of  the  most  challenging  tasks  was  determining  how  to  calculate  the  return  on  the  time  and 
resources  invested  in  the  LDAC  demonstration.  While  the  LDAC  resulted  in  increased  self-awareness 
and  motivation  for  the  majority  of  participants,  the  exact  value  of  participation  in  terms  of  increase  in 
performance  or  other  measureable  outcomes  of  interest  for  the  unit  or  the  Army  are  unknown.  No 
performance  data  were  captured  on  participants  after  the  assessments.  It  is  even  more  difficult  to 
directly  attribute  any  post-LDAC  increases  or  decreases  in  performance  of  participant  to  the  assessment 
center  experience  because  the  demonstration  did  not  use  a  control  or  comparison  group. 

Figure  7:  Summary  of  Time  Required  for  LDAC  Demonstration 
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Lessons  Learned 

Upon  the  completion  of  LDAC  activities,  staff,  role  players,  and  assessors  met  for  an  in-depth  after¬ 
action  review  on  the  development  and  administration  of  the  demonstration.  The  most  significant 
feedback  from  role-players  was  that  they  found  value  in  the  training  and  appreciated  the  organization  of 
the  LDAC.  They  also  highlighted  that  their  prior  military  experience  was  very  useful  and  helped  them 
better  deal  with  the  less  well-defined  parts  of  the  simulation.  However,  a  key  recommendation  for  future 
iterations  was  to  use  a  dedicated  and  trained  cadre  of  professional  role  players  rather  than  relying  on 
available  manpower  within  the  organization.  Assessors  recommended  having  additional  time  for  scoring 
the  simulations  and  having  different  assessors  evaluate  different  simulations.  Moreover,  assessors 
highlighted  the  need  for  cognitive  ability  and  personality  tests  that  were  developed  and  normed  using  an 
Army  population.  Participant  results  on  the  WPI-II  specifically  were  strangely  distributed  and  inconsistent 
with  the  characteristics  of  the  participating  CSMs.  One  possible  explanation  for  this  could  be  the  design 
of  the  WPI-II,  which  included  a  four-point  forced  choice  agreement  scale  and  item  wordings  that  may  not 
have  been  suited  for  the  Army  audience.  The  key  concern  among  assessors  was  that  the  six  month  lead 
time  was  too  short  to  allow  for  a  thorough  and  sufficient  developmental  process.  A  better  timeline  would 
have  included  multiple  iterations  and  validations  of  the  scenario  and  simulations  prior  to  execution. 
Moreover,  no  additional  staff  or  funding  was  provided  to  CAL  to  aid  in  or  offset  the  development  time 
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required  of  the  LDAC.  Due  to  these  resource  and  time  constraints,  it  was  not  possible  to  consistently 
apply  industry  standards  and  best  practices  related  to  developing  and  conducting  an  assessment  center. 
A  summary  of  the  LDAC  demonstration  functional  lines  as  compared  to  industry  best  practices  is  listed 
in  Table  5. 

Table  5:  Best  Practice  &  LDAC  Comparison 


Function 

Industry  Best  Practice 

Army  Demonstration 

Competency 

Modeling 

Competency  model  development  based  on  an 
in  depth  understanding  and  iterative  process  to 
identify  key  aspects  that  differentiate 
participants. 

Training  needs  analysis  for  CSEEC  allowed 
rapid  start  to  competency  model;  no 
performance  data  demonstrating  differentiation 
among  individuals  on  the  competencies. 

Assessors 

Trained  behavioral  psychologists,  multiple 
practice  opportunities. 

S  -  same  as  best  practice. 

Lead  Assessor 

Responsible  for  monitoring  center  oversight 
and  administration,  ensuring  quality  control. 

Has  limited  assessment  duties. 

One  assessor  had  dual  administration  and 
assessment  responsibilities. 

Psychometrician 

Trained  testing  specialist,  dedicated  to  scoring 
exercises,  administering  tests,  managing  tests. 

Assessors  and  support  team  filled  in  for  a 
dedicated  psychometrician. 

Center 

Administrator 

Dedicated  resource  responsible  for  managing 
the  execution  and  coordination  of  center 

V  -  same  as  best  practice. 

Role  players 

Use  consistent  cadre  of  professional  role 
players  with  senior  level  experience  with  the 
assessed  cohort.  At  least  one  dedicated  role 
player  per  simulation. 

Due  to  scheduling  constraints,  used  available 
role  players  with  prior  military  experience  but 
no  prior  formal  role-play  experience. 

Role  players  learned/executed  2  roles  rather 
than  just  1 . 

Testing 

Use  cognitive  ability  and  personality  tests  that 
have  been  validated  to  assessed  cohort 
populations  and  have  cohort-specific  norms 

Lack  of  available  Army-validated  measures; 
forced  use  of  private  sector  validated  tests. 

Schedule 

Scheduling  dedicated  to  operation  of 
assessment,  exercises  conducted  during 
normal  work  hours. 

Exercises  scheduled  around  gaps  in  CSMDP 
schedule  resulting  in  suboptimal  times. 

Participant 
Contact  time 

Three  or  more  simulations  with  in-depth 
feedback  that  occurs  over  10+  hours  over  span 
of  several  days. 

Major  phases  (e.g.  testing,  simulations  and 
feedback)  took  7  hours  over  3  Vfe  days,  all 
exercises  on  same  day. 

Coaching 

Participants  receive  developmental  support 
following  the  integrated  feedback  with  their  lead 
assessor. 

Participants  received  a  report  summarizing  their 
feedback,  lead  assessor  advised  on 
developmental  support  if  known. 

Facilities 

Dedicated  facilities  tailored  to  exercise 
scenario. 

Used  available  SCP  classrooms  without  any 
embellishment,  which  decreased  authenticity  of 
scenario. 

Participant 

Selection 

Individuals  self-select  or  are  nominated  for 
participation,  ensuring  personal  buy-in. 

Inconsistent  motivation  amongst  participants, 
some  of  whom  were  not  interested  in  nom-CSM 

Assessment  centers  with  a  greater  volume  of 

Per  Participant 
cost 

participants  would  have  a  lower  per-participant 
cost  as  upfront  development  costs  would  be 
distributed  across  a  larger  number.  Some 
standard  operating  costs  (e.g.  role  players, 
facilities,  etc.)  would  still  apply. 

Roughly  $8,500-$1 1 ,500  per  participant  for 
current  assessment  center  demonstration,  plus 
additional  opportunity  costs. 

Validation  & 
Continuous 
Improvement 

Structured  mechanism  for  validating  tests  and 
results  of  assessment  center  and  conducting 
continuous  improvement  to  practices 

Expert  review  of  part  of  the  demonstration 
(competency  model,  selected  tests,  exercises) 
and  participant  feedback  after 

ROI/  ROE 
Determination 

Value  estimates  should  be  determined  prior  to 
decisions  for  assessment  center. 

Demonstration  contributed  positive  information 
for  estimating  cost  and  participant  acceptance, 
but  no  follow-up  performance  data. 
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Appendix  A 

Final  Nominative  CSM  Competency  Model 

Intellect/Cognition 

JUDGMENT/DECISIONMAKING:  The  capacity  to  assess  situations  shrewdly  and  draw  sound  conclusions 
and  opinions,  make  sensible  decisions  and  reliable  guesses. 

■  Engages  in  thoughtful  assessment,  demonstrates  sound  judgment. 

■  Relates  and  compares  information  from  different  sources  to  identify  possible  cause-and-effect  relationships. 

■  Applies  critical  thinking  to  identify  faulty  logic  and  solution  pitfalls. 

■  Recognizes  the  need  to  gain  additional  information. 

■  Identifies  and  solves  problems  related  to  organizational  and  strategic  goals. 

■  Identifies  critical  issues  to  use  as  a  guide  in  making  decisions  and  taking  advantage  of  opportunities. 

■  Confidently  makes  decisions  in  the  absence  of  all  of  the  facts. 

Strategic-level  Thinking:  Strategic  thinking  is  a  deliberate  approach  to  thinking  about  a  situation 
and  what  to  do.  It  involves  thinking  broadly,  deeply  and  into  the  future:  Broad — seeing/making 
connections  across  the  organization  and  outside  the  Army.  Deep — deeply  questioning  problems,  their 
causes,  opportunities  to  improve,  and  solutions.  Future — shaping  solutions  far  into  the  future  so  the 
organization  can  implement  effective,  lasting  change.  Involves  thinking  about  the  complex  and  dynamic 
factors  that  go  well  beyond  the  typical  and  familiar  situations. 

■  Sees  problems  from  a  broad  perspective. 

■  Sees  organizations  and  what  they  address  in  a  connected,  systems  approach. 

■  Shifts  from  short-term  action  and  problem  solving  to  identifying  what  it  takes  for  long-term  success. 

■  Focuses  on  understanding  differently  and/or  deeply  to  create  meaning  from  a  complex  situation. 

■  Considers  the  unintended  consequences  of  actual  and  possible  actions. 

■  Identifies  high-risk  possibilities  under  rare  combinations  of  events. 

■  Applies  expert-level  knowledge  (tactical,  technical,  joint,  social,  cultural,  geopolitical)  to  derive  key  insights  or 
create  new  solutions. 

■  Has  a  comprehensive,  whole  government  (or  system)  view  that  takes  into  account  competing  demands  of 
diverse  stakeholders  (e.g.  inter-service  issues,  joint  staff,  external,  civilian,  congressional,  etc.). 

■  Considers  impact  of  external  factors,  world  political  situations,  US  political  climates,  etc.,  when  making 
recommendations. 

Cognitive  Flexibility/mental  Agility:  Models  a  flexible  mindset,  willing  to  be  flexible  in  approach, 
anticipates  and  scans  for  changing  conditions,  able  to  apply  fresh,  different  perspectives  to  problems. 

■  Demonstrates  willingness  to  consider  alternative  perspectives  to  resolve  difficult  problems. 

■  Tendency  to  anticipate  or  adapt  to  uncertain  or  changing  situations. 

■  Recognizes  when  standard  operating  procedures  will  not  produce  optimal  results. 

■  Adjusts  previous  plans  to  deal  with  a  changing  situation.  Engages  in  multiple  approaches  when  assessing 
situations,  generating  courses  of  action,  and  evaluating  them. 

■  Thinks  through  outcomes  when  current  decisions  or  actions  are  not  producing  desired  effects. 

■  Keen  ability  to  learn  with  a  corresponding  mindset  that  learning  is  important  and  highly  valued  that 
inquisitiveness  leads  to  greater  knowledge  and  that  learning  from  mistakes  is  acceptable  and  encouraged. 

■  Displays  comfort  working  in  complex  situations  with  no  preset  rules  and  no  known  outcomes. 
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Advising 


Advise/Influence:  Advises  the  senior  leaders  and  command  teams  on  enlisted  and  noncommissioned 
areas.  Uses  appropriate  methods  of  influence  to  energize  others,  ranging  from  compliance  to 
commitment  (pressure,  legitimate  requests,  exchange,  personal  appeals,  collaboration,  rational 
persuasion,  apprising,  inspiration,  participation,  and  relationship  building). 

■  Influences  beyond  his  or  her  direct  line  of  authority  and  beyond  chains  of  command  to  include  unified  action 
partners.  In  these  situations,  uses  indirect  means  of  influence:  diplomacy,  negotiation,  mediation,  arbitration, 
partnering,  conflict  resolution,  consensus  building,  and  coordination. 

■  Understands  sphere,  means  and  limits  of  influence  in  boundary  spanning  contexts. 

■  Outside  Army  boundaries,  strategic  leaders  have  a  role  as  integrator,  alliance  builder,  negotiator,  and  arbitrator. 

■  Identifies  individual  and  group  interests. 

■  Establishes  internal  and  external  networks  to  achieve  strategic  goals.  Proactive  in  creating  relationships. 

■  Demonstrates  effective  us  of  indirect  influence  techniques  (diplomacy,  negotiation,  mediation,  arbitration, 
partnering,  conflict  resolution,  consensus  building,  and  coordination). 

■  Establishes  trust  to  extend  influence  outside  the  chain  of  command.  Proactively  builds  and  maintains  alliances 
to  benefit  the  organization. 

■  Is  effective  in  dealing  with  officers,  civilians,  congressional  members,  and  the  press. 

■  Advises  leaders  on  the  cultural  considerations  for  planning  and  executing  missions. 

■  Motivates,  inspires,  and  influences  others  to  take  initiative,  work  toward  a  common  purpose,  accomplish  critical 
tasks,  and  achieve  organizational  objectives. 

■  Provides  purpose,  motivation,  and  inspiration  to  guide  others  toward  mission  accomplishment;  ensures 
subordinates  understand  and  accept  direction;  empowers  and  delegates. 

■  Enforces  standards,  reinforces  the  importance  of  standards,  recognizes  when  standards  are  being  met  and 
addresses  appropriately. 

COMMUNICATION:  Communicates  effectively  by  clearly  expressing  ideas  and  actively  listening  to  others. 

■  Informs  others  of  key  information. 

■  Demonstrates  proficient  interaction  with  others,  demonstrates  good  interpersonal  awareness  and  effectively 
adjusts  behaviors  when  interacting  with  others. 

■  Understands  others. 

■  Understands  and  effectively  communicates  vision,  intent,  and  strategy  with  internal  and  external  audiences. 
Communicates  commander’s  intent  and  talking  points  to  the  press. 

■  Uses  verbal  and  nonverbal  means  to  maintain  listener  interest.  Adjusts  information-sharing  strategy  based  on 
operational  conditions.  Ensures  information  dissemination  to  all  levels  in  a  timely  manner.  Avoids 
miscommunication  through  verifying  a  shared  understanding. 

■  States  goals  to  energize  others  to  adopt  and  act  on  them. 

■  Uses  logic  and  relevant  facts  in  dialogue;  expresses  well-organized  ideas. 

■  Determines,  recognizes,  and  resolves  misunderstandings. 
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Appendix  B 


LDAC  Cognitive  Ability  &  Personality/Dispositional  Tests 

Cognitive  Ability  Tests 


Description 

Administration 

Factors 

Raven’s  Standard 

Progressive 

Matrices 

Nonverbal  assessment  tool  designed  to 
measure  an  individual’s  ability  to 
perceive  and  think  clearly,  make 
meaning  out  of  confusion,  and  formulate 
new  concepts  when  faced  with  novel 
information. 

28  items,  40  min  timed 
Proctored  during  LDAC 
Norm:  General 

Population 

Abstract 
reasoning,  fluid 
intelligence 

Army  Critical 
Thinking  Test 

Designed  to  assess  critical  thinking,  the 
“the  volitional  use  of  reasoning  and 
integration  skills,  in  response  to  new 
information  or  a  new  context,  to  form  a 
conclusion  that  will  guide  one’s  behavior, 
knowledge,  expertise,  and/or  emotion.” 

28  items  (MC  &  short 
answer) 

40  min  timed 

Proctored  during  LDAC 
Norm:  n/a 

Analysis 

Inference 

Conjecture 

Integration 

Dispositional  &  Personality  Tests 


Description 

Administration 

Factors 

Workplace 

Personality 

Inventory 

Designed  to  measure  sixteen  work  styles  or  work- 
related  personality  traits  that  are  important  to  job 
success  in  a  wide  range  of  occupations.  Originally 
developed  based  on  the  work  styles  model  developed 
by  the  US  Dept  of  Labor  for  Occupational  Information 
Network  (0*NET®).  Feedback  report  included  an 
interpretive  and  development  guide  for  participants. 

192  items 

4-pt  Likert 
agreement  scale 

Unproctored, 
completed  prior  to 
LDAC 

Norm:  General 

Labor 

Achievement  - 
achievement/effort, 
persistence,  initiative 

Social  influence  -  leadership 
orientation,  social  orientation 
Interpersonal  -  cooperation, 
concern  for  others 
Self-adjustment  -  self-control, 
stress  tolerance, 
adaptability/flexibility 
Conscientiousness  - 
dependability,  attention  to 
detail,  rule  following 

Practical  intelligence  - 
innovation,  analytical  thinking, 
independence 

Deep 

Learning 

Orientation 

Battery 

Self-report  assessment  designed  to  measure 
individual  learning/meta-cognitive  strategies  and 
tendencies  to  engage  in  deep  learning.  Deep  learning 
occurs  when  the  learner  relates  new  concepts  to 
existing  experience,  distinguishes  between  new  ideas 
and  existing  knowledge,  and  evaluates  and 
determines  key  themes  and  concepts.  Because  the 
material  is  learned  at  a  deeper  level,  it  is  more  likely 
to  be  retained.  Additionally,  learners  are  better  able  to 
apply  the  information  in  a  changing  context,  relate 
new  information  to  the  real  world,  and  reinterpret 
knowledge  in  the  face  of  change.  The  deep  learning 
process  involves  both  learning  and  monitoring 
strategies.  Deep  learning  can  enhance  leader’s 
adaptability,  meaning  that  they  will  be  better  able  to 
deal  with  changing  requirements. 

44  items 

5-pt  Likert 
agreement  scale 
Unproctored, 
completed  prior  to 
LDAC 

Norm:  n/a 

Mastery  Orientation 

Performance  Orientation 
Elaborative  Processing 

Need  for  closure 
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Appendix  C 
Influence  Test 

The  Influence  Test  for  the  LDAC  was  developed  from  four  tests  developed  for  separate  purposes,  by 
different  developers,  and  at  different  times.  The  resulting  hybrid  test  has  six  parts  from  the  four  original 
tests.  Leaders  influence  others  toward  a  goal  by  getting  their  compliance  or  commitment.  Influence  is 
getting  people  to  think  or  do  like  what  the  influencer  wants  them  to.  Compliance  is  influence  operating 
based  on  conforming  to  a  requirement  or  a  demand,  while  commitment  is  influence  that  operates  on 
creating  a  willing  dedication  to  a  requirement  or  cause.  Compliance  is  appropriate  for  short-term, 
immediate  requirements  and  when  the  target  of  influence  is  unfamiliar  with  the  task.  Commitment 
produces  longer  lasting  effects  and  good  will  or  trust.  Resistance  is  when  the  target  of  influence  objects  to 
adopting  what  the  influencer  wants.  (For  more  information  see  ADRP  6-22,  paras.  6-1 -6-4.) 

Understanding  Influence 

Part  1 .  Understanding  instances  of  the  categories  will  help  military  leaders  be  better  at  understanding  and 
applying  influence  and  better  at  overcoming  resistance. 

Part  2.  Distinguishing  between  different  types  of  influence  techniques  will  help  military  leaders  understand 
better  understand  what  they  are  and  how  to  use  them.  (See  ADRP  6-22,  paras.  6-5  -  6-14.] 

Part  3.  Knowing  when  the  influence  techniques  will  be  most  effective  will  help  leaders  pick  the  best 
technique  for  the  right  situation. 

Part  4.  Three  vignettes  are  given  for  the  leader  to  review  and  identify  a)  the  resistance  that  was  present, 
b)  considerations  in  determining  influence,  and  c)  selection  of  a  good  influence  technique.  The  3  vignettes 
were  scored  against  a  rubric  created  with  the  assistance  of  the  SME  trainers  for  the  FSF  training.  For 
each  question,  scorers  assigned  between  1-8  points,  with  1-4  representing  an  ineffective  level  of 
performance  and  5-8  representing  an  effective  level  of  performance. 

Situation  Judgment  Test  (CAL  developed  leadership  test) 

Part  5.  The  Situational  Judgment  Test  (SJT)  items  were  selected  from  three  pools  of  items  from  3 
versions  of  the  CAL  SJT  tests  for  field  grade  officer,  company  grade  officer,  and  NCO.  The  items  were 
revised  for  the  LDAC  CSM  assessments  to  be  relevant  to  senior  NCOs.  The  situations  and  response 
options  were  developed  and  tested  to  distinguish  a  leader’s  performance  along  effectiveness  on  the 
Army’s  leadership  competencies.  Consensus  scoring  among  practitioners  was  used  to  identify  the  most 
effective  and  least  effective  responses.  Scores  on  individual  items  are  based  on  the  consensus  rank 
ordering  of  most  and  least  effective  options.  An  item  score  is  the  average  percentage  of  the  least  and 
most  effective  selection.  An  overall  average  is  provided  for  the  7  SJT  items  used  in  this  assessment.  72- 
100  Army  leaders  participated  in  a  content  validation,  4  of  the  7  items  have  comparison  values  as  follows. 

NC93 — leads  others:  Most  50.0%,  Least  53.0%,  Avg  51 .5% 

CG4 — communicates:  Most  50.0%,  Least  53.0%,  Avg  51.5% 

FG57 — leads  others/extends  influence:  Most  32.0%,  Least  66.0%,  Avg  49.0% 

FG8 — gets  results:  Most  60.4%,  Least  54.2%,  Avg  57.3% 

Applying  Influence  Strategies 

Part  6.  This  part  of  the  activity  is  part  of  an  ARI-developed  self-reflection  exercise.  Participants  receive  a 
score  that  indicates  how  likely  they  said  they  would  perform  various  influence  tactics  in  six  scenarios.  The 
six  were  selected  based  on  how  well  they  could  be  revised  to  apply  to  senior  NCOs.  A  participant’s  Likert- 
type  likelihood  answers  are  combined  to  reflect  a  likelihood  total  for  each  of  12  measured  influence  tactics 
(similar  to  the  ADRP  6-22  techniques,  but  not  exact).  The  explanation  of  the  techniques  used  by  ARI  is 
provided  on  the  following  page.  The  likelihood  of  using  the  3  strongest  commitment  strategies  or  the  3 
strongest  compliance  strategies  is  also  shown  on  the  score  sheet. 
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Scoring  Procedure 

Parts  1  through  3,  representing  a  knowledge-level  test  on  influence,  were  scored  using  a  key  of  correct 
answers.  There  were  a  total  of  32  items  across  these  three  parts.  A  score  of  0  to  32  was  possible  for 
Parts  1  through  3.  Eighty-six  leaders  who  had  been  administered  the  test  in  2009  scored  an  average  of 
25  out  of  32  points. 

Part  4  with  the  short  answer  format  used  a  content  rubric  that  granted  points  for  correct  inclusion  and 
assessment  of  information.  The  rubric  awarded  from  1  (most  ineffective)  to  8  (most  effective)  points  for 
each  question.  Scoring  was  done  by  a  subject  matter  expert  who  had  sample  answers  and  scores  from  a 
previous  administration  to  train  on  before  scoring.  The  vignettes  on  part  4  could  have  a  final  score  of  3  to 
24  for  each  vignette. 

Part  5  was  scored  comparing  the  two  answers  (most  and  least  effective)  to  a  consensus  score  based  on 
expert  opinion  and  sample  data.  Each  set  of  response  options  was  compared  to  a  rank  ordering.  For 
example  if  a  participant  picked  response  ‘C’  to  be  the  best  answer  for  situation  1 ,  the  participant  obtained 
7  out  of  8  possible  points  because  response  ‘C’  was  actually  the  second  best  answer.  Points  were 
combined  for  best  and  least  effective  actions  and  then  averaged  across  the  seven  situations.  The  best 
possible  raw  score  was  14,  the  worst  possible  raw  score  was  2.08.  A  spreadsheet  with  a  look-up  table  of 
answers  and  calculations  was  used  as  a  scoring  aid.  A  final  percentage  score  from  0%  to  100%  correct 
was  assigned  to  Part  5  of  the  influence  exercise. 

Part  6  was  scored  by  summing  the  likelihood  rating  that  the  participant  gave  each  influence  tactic 
presented  for  each  vignette.  Influence  techniques  were  not  equally  represented  across  the  vignettes, 
some  were  represented  two  times  and  others  represented  up  to  five  times  across  the  six  vignettes.  A 
spreadsheet  was  used  to  calculate  the  score. 

Scores  for  the  influence  exercise  were  converted  to  a  1  to  5  metric  common  to  the  other  exercises  in  the 
nominative-CSM  assessment  center  (see  Appendix  C  for  detailed  description  of  influence  test  scoring). 
Similar  to  the  protocol  for  the  cognitive  and  non-cognitive  assessments,  participants  received  a  score 
ranging  from  1  to  5  on  Parts  1  through  3,  Part  4  and  Part  5.  Transformed  scores  were  based  on  equal 
division  of  the  range  of  scores.  Parts  1,2,3  (the  knowledge  test),  and  two  of  the  Part  4  vignettes  were 
combined  to  represent  the  advise  and  influence  competency.  The  third  vignette  of  Part  4  and  the 
situational  judgment  test  (Part  5)  were  combined  to  represent  the  judgment  and  decisionmaking 
competency.  Part  6  represented  style  of  influence,  and  results  were  reserved  for  use  as  qualitative 
feedback. 
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Influence  Tactic 


Description 


Appeal  to  Duty 
and/or  Morality 

Involves  making  appeals  to  the  counterpart’s  conscience  and  desire  to  do  the  right  thing.  For 
example,  the  leader  might  tell  a  counterpart  that  taking  an  action  is  their  duty  or  moral  obligation. 

This  tactic  may  sometimes  use  guilt  as  a  motivator,  but  also  may  rely  on  a  counterpart’s  desire  to 
follow  norms  and  social  conventions. 

Inspirational 

Appeal 

Inspirational  appeal  is  a  tactic  in  which  the  leader  attempts  to  motivate  a  counterpart  to  take  action  by 
appealing  to  an  individual’s  values,  ideals,  and  aspirations.  In  this  instance,  the  leader  is  attempting 
to  persuade  the  counterpart  to  take  action  out  of  a  higher  calling.  While  appealing  to  one’s  sense  of 
duty  may  activate  the  motivation  to  avoid  guilt  or  to  comply  with  social  norms  currently  in  place,  an 
inspirational  appeal  attempts  to  persuade  a  counterpart  by  generating  enthusiasm  for  something 
selfless  and  noble 

Rational 

Persuasion 

The  leader  uses  logic  and  facts  to  explain  to  a  counterpart  why  a  course  of  action  should  be  adopted 

Collaboration 
(not  assessed) 

The  leader  offers  to  provide  assistance,  resources,  or  other  forms  of  leader-counterpart  partnership 
to  entice  the  counterpart  to  behave  or  act  in  a  certain  way.  Collaboration  can  be  a  useful  strategy  for 
leader  because  it  gives  a  counterpart  ownership  over  the  course  of  action  and  potentially  can 
mitigate  a  counterpart's  concerns  about  responsibility  and  limited  resources. 

Establishing 

Rapport  & 

Creating  Positive 
Feelings 

When  a  leader  focuses  on  building  rapport,  creating  goodwill  with  a  counterpart,  or  communicating 
an  understanding  of  the  counterpart's  point  of  view  before  making  a  request,  the  leader  is  using 
rapport  building  as  an  influence  strategy.  In  FM  6-22,  this  influence  technique  is  referred  to  as 
relationship  building. 

Use  Rank  or 
Authority 

When  a  leader  exercises  the  power  of  authority  associated  with  his  or  her  rank  or  position,  the  leader 
is  using  rank  and  authority  as  a  means  for  persuading  others.  In  FM  6-22,  this  strategy  is  referred  to 
as  making  a  legitimate  request. 

Use  Pressure, 
Threats,  or 

Warnings 

When  a  leader  demands  that  a  counterpart  adopt  a  course  of  action  and  strongly  emphasizes  that 
negative  consequences  will  result  if  the  counterpart  fails  to  take  that  course  of  action,  the  leader  is 
using  pressure  tactics.  Pressure  tactics  may  take  the  form  of  an  overt  threat,  such  as  removing 
advisor  support,  or  may  be  more  subtle,  such  as  providing  a  warning  that  something  bad  will  happen 
in  the  future  as  a  result  of  failing  to  adopt  the  advisor’s  course  of  action 

Coalition  Tactics 

When  a  leader  uses  the  involvement  or  support  of  others  to  persuade  a  counterpart  to  comply  with  a 
request,  then  the  leader  is  using  coalition  tactics.  It  should  be  noted  that  the  leader  might  bring  his  or 
her  “coalition”  to  a  meeting  or  the  leader  may  merely  say  that  other  people  are  “on  board”  with  the 
advisor’s  point  of  view 

Use  Negative 
Emotion 

When  a  leader  demonstrates  a  negative  emotion  such  as  anger,  fear,  or  sadness  to  persuade  a 
counterpart  to  adopt  a  course  of  action,  the  leader  is  using  negative  emotions  as  an  influence  tactic. 
Using  negative  emotions  can  sometimes  be  used  as  a  form  of  intimidation.  For  example,  anger  may 
be  used  to  amplify  the  use  of  pressure  and  threats.  In  other  instances,  negative  emotion  may  be 
used  to  match  the  mood  of  the  counterpart  to  build  rapport  and  camaraderie.  Sometimes  negative 
emotion  may  simply  be  used  to  convey  information  about  the  importance  of  the  situation  and  the 
necessity  to  take  action 

Apprising  Tactics 
(not  assessed) 

When  a  leader  explains  how  compliance  will  personally  benefit  his  or  her  counterpart,  the  leader  is 
apprising  the  counterpart  of  the  positive  benefits  of  taking  a  course  of  action  (e.g.,  taking  action  will 
help  to  develop  one’s  expertise).  Unlike  exchange  tactics,  the  resulting  personal  benefits  to  the 
counterpart  are  outside  the  control  of  the  advisor 

Exchange  Tactics 

When  a  leader  offers  something  of  value  in  exchange  for  compliance  with  the  request,  the  leader  is 
using  some  form  of  exchange  to  persuade  the  counterpart  to  take  action.  Unlike  apprising  tactics, 
exchange  tactics  focus  on  the  exchange  of  rewards  or  positive  benefits  that  are  within  the  control  of 
the  advisor 

Personal  Appeal 

When  an  leader  attempts  to  persuade  a  counterpart  by  appealing  to  the  counterpart’s  sense  of 
friendship  or  loyalty,  the  leader  is  making  a  personal  appeal 

Use  an  Indirect 
Approach 

A  leader  is  using  an  indirect  approach  by  hinting  to  the  counterpart  or  using  other  indirect 
suggestions  to  take  action 

Pair  Requests 
Strategically 

The  social  psychology  literature  indicates  that  individuals  can  use  successive  requests  in  a  way  that 
makes  compliance  more  likely.  One  well  known  strategic  pairing  is  called  the  foot-in-the-door 
principle.  In  this  principle,  the  leader  might  make  a  small  request  to  make  the  counterpart 
comfortable  with  compliance.  The  leader  may  follow  up  with  successively  greater  requests  until  the 
advisor’s  overall  objective  is  achieved 
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